New approaches to operating system security extensibility

This dissertation proposes new approaches to commodity computer operating system (OS) access control extensibility that address historic problems with concurrency and technology transfer. Access control extensibility addresses a lack of consensus on operating system policy model at a time when security requirements are in flux: OS vendors, anti-virus companies, firewall manufacturers, smart phone developers, and application writers require new tools to express policies tailored to their needs. By proposing prin-cipled approaches to access control extensibility, this work allows OS security to be " designed in " yet remain flexible in the face of diverse and changing requirements. I begin by analysing system call interposition, a popular extension technology used in security research and products, and reveal fundamental and readily exploited con-currency vulnerabilities. Motivated by these failures, I propose two security extension models: the TrustedBSD Mandatory Access Control (MAC) Framework, a flexible kernel access control extension framework for the FreeBSD kernel, and Capsicum, practical capabilities for UNIX. The MAC Framework, a research project I began before starting my PhD, allows policy modules to dynamically extend the kernel access control policy. The framework allows policies to integrate tightly with kernel synchronisation, avoiding race conditions inherent to system call interposition, as well as offering reduced development and technology transfer costs for new security policies. Over two chapters, I explore the framework itself, and its transfer to and use in several products: the open source Free-BSD operating system, nCircle's enforcement appliances, and Apple's Mac OS X and iOS operating systems. Capsicum is a new application-centric capability security model extending POSIX. Capsicum targets application writers rather than system designers, reflecting a trend towards security-aware applications such as Google's Chromium web browser, that map distributed security policies into often inadequate local primitives. I compare Capsicum with other sandboxing techniques, demonstrating improved performance, programmability, and security. This dissertation makes original contributions to challenging research problems in security and operating system design. Portions of this research have already had a significant impact on industry practice. 3 4 Acknowledgements Writing this dissertation would not have been possible without the support and encouragement of my family (especially my parents), friends, mentors, and colleagues, to whom I offer my sincerest thanks and appreciation. Ross Anderson, my supervisor, deserves a special note of thanks: he has been supportive throughout my less than typical path through Cambridge's PhD programme, giving me space to pursue a variety of interests, many related to my PhD …